Methods for the Classification of Data from Open-Ended Questions in Surveys

Which methods can we use to classify data from open-ended survey questions?

Can we leverage these methods to make empirical contributions to substantial questions?

• Background:
  • ongoing debates about which type of trust survey researchers are measuring with traditional survey items (i.e., equivalence debate cf. Bauer & Freitag 2018)
• Research Question:
  • How valid are traditional trust survey measures?
• Experimental Design:
  • block randomized question order where seven closed-ended questions are followed by open-ended follow-up probing questions

• Operationalization via two classifications: share of known vs. unknown others in associations (I), sentiment (pos-neu-neg) of assocations (II)
• Supervised classification approach:
  • 1. manual labeling of randomly sampled documents (n=[1,000,1,500])
  • 2. fine-tuning the weights of two BERT¹ models (base model uncased version), using the manually coded data as training data, to classify the remaining=[6,500/6,000]
  • accuracy²: 87% (I) and 95% (II)

• Background:
  • recent increase of voice-based response options in surveys due to mobile devices equipped with voice input technologies, smartphone surveys and speech-to-text technologies
• Research Question:
  • Are there differences in information content between responses given in voice and text formats?
• Experimental Design:
  • block randomized question order with open-ended and probing questions
  • random assignment into either the text or voice condition

• Operationalization via application of measures from information theory and machine learning to classify open-ended survey answers
  • number of topics, response entropy
  • response length

• Background:
  • conventional notion stating tha trust originates from informed, rational, and consequential judgments is challenged by the idea of an “affective-based” form of (political) trust
• Research Question:
  • Are individual trust judgments in surveys driven by affective rationales?
• Questionnaire Design:
  • closed-ended political trust question followed by open-ended probing question

• Operationalization via sentiment and emotion analysis

• Transcript-based

  • pysentimiento for sentiment recognition (Pérez et al. 2023)
  • zero-shot prompting with GPT-3.5

• Speech-based

  • SpeechBrain for Speech Emotion Recognition (Ravanelli et al. 2021)

• web surveys can be used to collect narrative answers that provide valuable insights into survey responses

• various modern developments (smartphone surveys, speech-to-text algorithms) can be leveraged to collect such data in innovative ways (e.g., spoken answers)

  • always consider challenges and objectives (i.e., in term of sample sizes and sample compositions)

• computational measures can inform ongoing debates in different fields by classifying open-ended answers from surveys, e.g.:

  • Study 1: equivalence debate in trust research
  • Study 3: cognitive-versus-affective debate in political trust research
  • Study 2: survey questionnaire design or item and data quality in general (e.g., associations, sentiment, emotions) (Study 1-3)

Facilitated accessibility and implementation of semi-automated methods.

• supervised models have been a standard in automated methods, but recent developments of large and general-aim pre-trained models (e.g., BERT) allow less resource-intensive fine-tuning

• For example, using only ~13% (1,000 documents from 7,500 in Study 1) documents for fine-tuning resulted in sufficient accuracy (i.e., 87%)

  • increasing the number of manually labeled documents can help in terms of accuracy (i.e., 92% in Study 1)
    • higher number of manual examples also improves the transparency of results: accuracy vs. transparency trade-off
      • start with simple methods and evaluate (e.g., Study 1, first Random Forest, only later BERT)

Increase in possibilities of fully automated methods (e.g., prompt engineering.

• fully automated methods, such as zero-shot prompting can keep up with fine-tuned versions of pre-trained models (e.g., pysentimiento, Study 2)
  • deciding on a suitable number of manual examples and for a method in general (e.g., fully automated (unsupervised) versus semi-automated (supervised, and finetuning) depends on resources such as expected difficulty, desired accuracy, available time and cost resources